Repository navigation
Add private-repo pre-flight, env-sourced gateway key, and the onboard-benchmark skill - #193
Merged
Merged
Conversation
Benchmarking a team's own repositories against a corporate model gateway hit two gaps. Private repositories already clone fine with ambient git credentials, but nothing checked them up front, so a bad token or a mistyped ref failed once per task, hours into a batch, as an opaque "git clone failed". And the endpoint API key could only be a literal in config/runner.yaml, which is the wrong place for a real gateway token. - preflight_check.py --check-repos resolves every task's repository and pinned ref and confirms each answers, using git ls-remote so it transfers no objects and costs seconds. It separates a missing credential, a rejected token, an untrusted internal CA, an unresolvable host and a tag that does not exist, and names the fix for each. Terminal prompting is disabled so an unattended check fails rather than hanging on a password prompt. run-e2e-benchmark.sh runs it before any task starts. - api_key_env names an environment variable holding the endpoint key. It wins over api_key and fails at config load when the variable is unset, rather than hours later inside the agent with an authentication error that never mentions it. Wired through the config model, the example YAML and both CLIs. - The per-run agent config files that carry the key (omp models.yml, pi models.json) are now written 0600 rather than umask-default, and --dry-run redacts the key it would otherwise print inside the claude settings JSON. - dataset/workshop-template.yaml is an annotated two-task starting point, built around pinning the release before a fix so the defect is present in the tree. - docs/benchmark-your-own-repo.md gains a section on private repositories and GitHub Enterprise Server: credential helpers, insteadOf token rewriting, deploy keys, the GIT_SSL_CAINFO and NODE_EXTRA_CA_CERTS pair, and where the key ends up per agent. Also replaces the pre-existing `-> "callable"` annotation in preflight_check.py with Callable[[str], str], so that file is now mypy-clean. 453 tests pass; ruff, mypy and bandit are clean.
Nothing guided someone from "we have models and repositories" to a first scored run on their own infrastructure. `setup-machine` installs dependencies and `benchmark` runs a model through a setup that already works; the gap between them is the whole of onboarding, and it is where a first attempt actually fails. New `onboard-benchmark` skill, in ten steps that each end in a proof: - Step 1 asks the question with the longest lead time, which is where the benchmark will run. Every task makes two calls that must both succeed from the same host: a clone from the code host and a model call to the gateway. Those usually sit on different networks, and a box reaching only one cannot run a single task. It must be verified from that machine, not from a diagram, and a subnet or peering change is not something to work around. - Step 2 reads the gateway's API shape and picks the harness that can speak it (OpenAI-compatible means omp or pi; Anthropic-compatible means claude), then settles whether its auth is a static key at all. - Steps 3 to 5 wire the agent, the judge and repository access, each proven with a real call, because a config file that looks right is not evidence. - Steps 6 to 10 author the dataset, run one smoke task, settle the cost basis, scale to a batch, and package the frontier as a swe-router skill. It also corrects the benchmark skill, which had drifted well past this change: - Agents: it documented `claude` and `pi` only. The orchestrator accepts `claude|pi|omp|kiro`, and omp, which produced the headline results, appeared nowhere in the file. - It claimed pi "has no native Amazon Bedrock mode, so it cannot run --provider bedrock". pi bundles the bedrock-runtime client, the harness has a PI_PROVIDER_BEDROCK constant, and two committed scripts do exactly that. - `--skill` was never documented as an input, while the text described `/swe2` throughout and the default is `swe3`. - Every artifact path was missing two directory levels: the layout is <model>/<harness>/<skill>/<scope>/<task>, so the judge and summarizer commands as written pointed at a tree the harness never writes to. - The default dataset was the 5-task v1 set rather than the 21-task v2 set the published results come from. - Added the repository-reachability pre-flight and `api_key_env` from the previous commit, and notes on the context_window requirement behind a custom base URL. The credential guidance now states that `credential.helper store` writes the token in plaintext, offers the environment-only and deploy-key alternatives, and says never to reach for GIT_SSL_NO_VERIFY when the fix is to trust the CA. Registered in AGENTS.md, README.md, docs/repository-structure.md and docs/benchmark-your-own-repo.md. All relative links verified to resolve.
Step 1 checked that the machine reaches the gateway and the code host, but not that it can reach a package index at all, which is what installing anything depends on. A locked-down host often has none of the three. Names PyPI (both pypi.org and files.pythonhosted.org, since wheels come from the second and allowlisting only the first fails partway through the install), registry.npmjs.org (the agent and the judge are npm packages), and the installer and OS sources. Covers the internal-mirror settings for uv and npm, and the NO_PROXY case where the gateway call gets routed to the proxy after everything installs cleanly. Four matching rows added to the failure-mode table.
People assume the first step needs dedicated infrastructure. Anywhere Python runs and dependencies can be installed works: an EC2 instance, a build or CI runner, or a development workstation or laptop. No GPU, since the models are hosted elsewhere.
Main added codex as a fifth coding agent (#162, #184), which needed reconciling beyond the textual conflicts. Textual conflicts, all in .claude/skills/benchmark/SKILL.md: - The agent input. Kept this branch's table, added a `codex` row, and kept main's Responses-API tool-parser warning, which is hard-won and still true. Did NOT keep main's claim that pi cannot run --provider bedrock: pi has a PI_PROVIDER_BEDROCK constant and two committed scripts that do exactly that, which is one of the drift fixes this branch makes. - The CLI pre-flight. Kept this branch's structure, added `codex exec --json` to the list of agent binaries. - The orchestrator command line. Took both additions: codex in the agent list and the --skill flag. One semantic conflict git could not see. Main's _build_codex_env sets OPENAI_API_KEY from config.api_key directly, which bypasses this branch's api_key_env and would send the "local" placeholder to a gateway that needs a real credential. Routed it through resolved_api_key(), so all six call sites that send the key now honour the environment variable. Also updated the onboard-benchmark skill, which said codex was only the judge: the harness-selection table now has a Responses-API row for it, and both that step and the judge step note that running --agent codex puts one tool on both sides of the run, so the judge model must differ from the model under test.
aarora79
added a commit
to shekharprateek/agentic-coding-harness-benchmarks
that referenced
this pull request
Sep 29, 2026
Main gained the private-repo pre-flight, the env-sourced gateway key and the onboard-benchmark skill (aarora79#193) while this branch was open. Textual conflicts, all three in .claude/skills/benchmark/SKILL.md, where this branch and aarora79#193 both rewrote the agent documentation: - The agent input. Took aarora79#193's table form and added a `strands` row, rather than keeping the older prose list. The row records what the orchestrator actually enforces: strands runs on either provider, and unlike every other agent it has no CLI binary. Dropped the stale claim that pi cannot run --provider bedrock, which aarora79#193 corrected. - The CLI pre-flight. Took aarora79#193's structure and kept this branch's strands fact as a following paragraph: there is no binary to find on PATH, the SDK runs in-process, and the orchestrator installs the optional dependency group with `uv sync --group strands`. - The orchestrator command line. Took both additions, strands in the agent list and aarora79#193's --skill flag. One semantic conflict git could not see. `_build_strands_env` sets the Strands API key from `config.api_key` directly, which bypasses the `api_key_env` support aarora79#193 added and would send the "local" placeholder to a gateway that needs a real credential. Routed through `resolved_api_key()`, matching the other six call sites. Verified: with api_key_env set, the strands env now carries the token from the environment rather than the placeholder. Also added strands to the harness-selection table in the onboard-benchmark skill, which aarora79#193 introduced and this branch had no way to know about, so the skill does not silently omit an agent.
Merged
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Why
Benchmarking a team's own repositories against a corporate model gateway hits two gaps that only show up off the published paths.
Private repositories already clone correctly:
_clone_repoand the judge'sclone_repo_at_refboth shell out to plaingit clonewith the inherited environment, so an ambient credential helper or aninsteadOfrewrite works with no code change. What was missing is a check. The harness clones per task, so a bad token or a mistyped ref failed once per task, hours into a batch, as an opaquegit clone failed.The endpoint API key could also only be a literal in
config/runner.yaml. That is right for a self-hosted vLLM server, which ignores the value, and wrong for a gateway token.What changed
Repository reachability pre-flight.
preflight_check.py --check-reposresolves every task's repository and pinned ref and confirms each answers, usinggit ls-remoteso it transfers no objects and costs seconds. It separates a missing credential, a rejected token, an untrusted internal CA, an unresolvable host and a tag that does not exist, and names the fix for each.GIT_TERMINAL_PROMPT=0means an unattended check fails rather than hanging on a password prompt.run-e2e-benchmark.shruns it before any task starts.Measured against the real dataset it is 0.2s for one repo, and it correctly diagnoses all three failure modes:
api_key_env. Names an environment variable holding the endpoint key. It wins overapi_keyand fails at config load when the variable is unset, rather than hours later inside the agent with an authentication error that never mentions it. Wired through all three layers per the AGENTS.md config checklist: the Pydantic field,runner.example.yaml, and a--api-key-envflag on bothrun-swe-headless.pyandrunner_config.py.dataset/workshop-template.yaml. An annotated two-task starting point, built around pinning the release before a fix so the defect is actually present in the cloned tree. Validates as-is.Docs.
docs/benchmark-your-own-repo.mdgains a section on private repositories and GitHub Enterprise Server: credential helpers,insteadOftoken rewriting, deploy keys, theGIT_SSL_CAINFOandNODE_EXTRA_CA_CERTSpair (two variables, two separate failures, one root cause), and the pin-the-release-before-the-fix rule promoted out of a tip.Security findings from the gate, and their fixes
The
security-checkgate flagged two issues created by letting a real token flow down a path that previously only ever carried thelocalplaceholder. Both are fixed in this PR._write_omp_configand_write_pi_models_jsonemit the endpointapiKeyinto a per-runmodels.yml/models.jsonviawrite_text, so umask left them0644. Both are nowchmod 0o600(pattern Gate the benchmark on a >=200K context window #7: a machine-written credential belongs in an owner-only file).--dry-runprinted the key. It prints the full agent command, and on the claude path that command carries--settingsinline JSON containingANTHROPIC_API_KEYandapiKeyHelper: echo <token>. Harmless atlocal, a credential leak into any redirected dry-run output at a real token._redact_api_keynow masks it, and deliberately leaves thelocalplaceholder alone so paths like~/.local/binare not mangled.One residual risk, documented rather than fixed. On the
claudepath the settings JSON is a command-line argument, so a real key is visible in the process table to any other local user. Claude Code requires a token source even against an endpoint that ignores it, and restructuring how settings are passed would touch the hot path every published result depends on. The docs now state this per-agent (ompandpikeep the key in a0600file and are unaffected) and point atsettings_fileas the mitigation. Worth a follow-up issue if anyone runs the harness on a shared box.Verification
api_key_env, 5 for the two security fixes.ruff check,ruff format --check, andbandit -r scripts/clean (0 issues at every severity).mypyclean ondataset_loader.py,runner_config.pyandpreflight_check.py. The last was not previously in the CI mypy target and had two pre-existing errors from a-> "callable"annotation directly above the new code; replaced withCallable[[str], str]so the file is clean.mcp-gateway-registryrepo and against all three failure modes, and a--dry-runwith a token in the environment confirming zero occurrences of the secret in the output.bash -non the modified orchestrator. The new shell step mirrors the existingpreflight_check.pycall two lines below it; it was not exercised end to end because this box has no gateway to run a benchmark against.Not in scope
Nothing outstanding. An earlier version of this PR said codex was the judge and not a harness;
mainadded it as a fifth agent (#162, #184) while this branch was open, and the merge below reconciles that.Second commit: the
onboard-benchmarkskill, and drift inbenchmarkWhy a new skill
Nothing guided someone from "we have models and repositories" to a first scored run on their own infrastructure.
setup-machineinstalls dependencies;benchmarkruns a model through a setup that already works. The gap between them is the whole of onboarding, and it is where a first attempt actually fails.onboard-benchmarkcovers it in ten steps, each ending in a proof rather than a filled-in config file:omporpi; Anthropic-compatible meansclaude), then settle whether its auth is a static key at all.3-5. Wire the agent, the judge and repository access, each proven with a real call. A successful
curldoes not prove the CLI works, and an unconfiguredcodexignores ambient credentials entirely.6-10. Author the dataset, run one smoke task, settle the cost basis, scale to a batch, package the frontier as a
swe-routerskill.It carries the honest timeline from measured runs: 110 to 449 minutes per 21-task model, so a first session produces one scored task and a working pipeline.
Drift corrected in
benchmarkChecking whether that skill needed my pre-flight change turned up more than expected. Anyone following it today would be told the headline harness does not exist and pointed at folders that are not there.
claudeandpionlyclaude|pi|omp|kiro.ompappeared zero times, and it produced the published frontier--provider bedrock"PI_PROVIDER_BEDROCK = "amazon-bedrock"plus two committed scripts that do exactly that--skillundocumented; text described/swe2throughoutswe3is the defaultswe-benchmark-data/{model-slug}/<repo>/<task>/Also added there: the repository-reachability pre-flight and
api_key_envfrom the first commit, a note thatcontext_windowis required behind a custom base URL, and a pointer toonboard-benchmarkfor a first run.Security note on the new prose
Reviewing my own guidance caught one thing worth fixing: the credential instructions recommended
credential.helper storewithout saying it writes the token in plaintext to~/.git-credentials. Both the skill and the docs now say so, offer the environment-onlyinsteadOfform and deploy keys as alternatives, and state explicitly thatGIT_SSL_NO_VERIFYis never the fix for a TLS error (trust the CA instead).Registration and verification
Registered in
AGENTS.md,README.md,docs/repository-structure.mdanddocs/benchmark-your-own-repo.md. Every relative link in both skill files verified to resolve.Docs and skills only in this commit, no Python or shell touched. Verified locally: all 7
pre-commithooks pass on every file changed in the branch,ruff check,ruff format --checkandmypyclean, 453 tests pass.Merged
main(13 commits), and what that neededmainadded codex as a fifth coding agent (#162, #184) while this branch was open. That is the change that mattered; the rest merged cleanly.Textual conflicts, all in
.claude/skills/benchmark/SKILL.md:codexrow, and kept main's Responses-API tool-parser warning, which is hard-won and still true. Deliberately did not keep main's claim that pi cannot run--provider bedrock: pi has aPI_PROVIDER_BEDROCKconstant and two committed scripts that do exactly that, and correcting it is one of the drift fixes this PR makes.codex exec --jsonto the agent binaries.--skill.One semantic conflict git could not see. Main's
_build_codex_envsetsOPENAI_API_KEYfromconfig.api_keydirectly, which bypasses this branch'sapi_key_envand would send thelocalplaceholder to a gateway that needs a real credential. Routed throughresolved_api_key(), verified live: withapi_key_env: WS_TOKENset,_build_codex_envnow returns the token from the environment. All six call sites that send the key honour it.The
onboard-benchmarkskill said codex was only the judge. Its harness-selection table now has a Responses-API row for codex, and both that step and the judge step note that--agent codexputs one tool on both sides of the run, so the judge model must differ from the model under test.One test failure during the merge was an artifact, not a defect.
test_committed_models_json_matches_the_generatorfailed while the merge was staged but uncommitted:measured_oncomes fromgit log -1 --format=%cson the frontier JSON, sogit logstill saw only this branch's history (2026-09-09) while the working-tree file was main's (2026-09-10). Committing the merge made main's commit reachable and the test passes. Confirmed the test passes onorigin/mainalone in a clean worktree first, so it was never a pre-existing failure.Post-merge verification: 498 tests pass,
ruff check,ruff format --check,mypyandbanditclean (0 findings at every severity),bash -non the orchestrator. All 5 CI checks green on the merge commit.